Automated Pyramid Scoring of Summaries using Distributional Semantics
نویسندگان
چکیده
The pyramid method for content evaluation of automated summarizers produces scores that are shown to correlate well with manual scores used in educational assessment of students’ summaries. This motivates the development of a more accurate automated method to compute pyramid scores. Of three methods tested here, the one that performs best relies on latent semantics.
منابع مشابه
Evaluating Content Selection in Human- or Machine-Generated Summaries: The Pyramid Scoring Method
From the outset of automated generation of summaries, the diÆculty of evaluation has been widely discussed (e.g., [10] [25]). Despite many promising attempts, we believe it remains an unsolved problem. Here we present a method for scoring the content of summaries of any length against a weighted inventory of content units, which we refer to as a pyramid. Our method is derived from empirical ana...
متن کاملPEAK: Pyramid Evaluation via Automated Knowledge Extraction
Evaluating the selection of content in a summary is important both for human-written summaries, which can be a useful pedagogical tool for reading and writing skills, and machinegenerated summaries, which are increasingly being deployed in information management. The pyramid method assesses a summary by aggregating content units from the summaries of a wise crowd (a form of crowdsourcing). It h...
متن کاملNouveau-ROUGE: A Novelty Metric for Update Summarization
An update summary should provide a fluent summarization of new information on a timeevolving topic, assuming that the reader has already reviewed older documents or summaries. In 2007 and 2008, an annual summarization evaluation included an update summarization task. Several participating systems produced update summaries indistinguishable from humangenerated summaries when measured using ROUGE...
متن کاملDeep Learning for Automatic Summary Scoring
Automatic summary scoring is used very often by summarization system developers to test different algorithms and to tune their system. We have developed a new approach based on representation learning, using both unsupervised and supervised learning components, to score a summary based on examples of manually evaluated summaries. Our deep learning approach greatly surpassed ROUGE in terms of co...
متن کاملAutomation of Summarization Evaluation Methods and their Application to the Summarization Process
Summarization is the process of creating a more compact textual representation of a document or a collection of documents. In view of the vast increase in electronically available information sources in the last decade, filters such as automatically generated summaries are becoming ever more important to facilitate the efficient acquisition and use of required information. Different methods usi...
متن کامل